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Executive Summary 

The Office of Shared Accountability (OSA) conducted an evaluation of the implementation of 
the second year (2011-2012) of the Science, Technology, and Engineering Leadership Program 
(STELP) in Montgomery County Public Schools (MCPS). The study was requested by the 
Office of Curriculum and Instructional Programs (OCIP). Funding for STELP, including the 
evaluation study, is provided by a grant from the Howard Hughes Medical Institute (HHMI) to 
MCPS. 


Background and Evaluation Questions 

The goal of STELP is to improve instruction in science, technology, and engineering and help 
students achieve science, technology, engineering, and mathematics (STEM) literacy 
(MCPS, 2010). This evaluation report addresses the second year of STELP. The focus of this 
evaluation was on implementation of the program in terms of continued training of a group of 
teacher leaders to develop online professional development modules for other MCPS science, 
technology, and engineering (STE) educators to view. Year two included both new and returning 
teacher leaders and built upon the professional development modules created in year one. The 
evaluation addressed the following questions: 

1. What was the context of the Science, Technology, and Engineering Leadership Program? 

2. To what extent was the training of teacher leaders implemented as planned? 

3. What was the impact of the STELP training sessions on teacher leaders? 

4. When and how were the online products made available to MCPS teachers? 


Summary of Key Findings 

1. What was the context of the Science, Technology, and Engineering Leadership Program? 

Characteristics of the participating teacher leaders. A total of 47 teachers participated in 
the second year of STELP. Of the 47 teachers, 30 participated in the first year of STELP as well. 
The participants represented a variety of positions in both elementary and secondary schools. 
Almost everyone had more than five years teaching experience (98%), and the majority reported 
having experience teaching science (92%). Teachers who participated in STELP were from 
22 elementary schools, 15 middle schools, and 1 high school. 

2. To what extent was the training of teacher leaders implemented as planned? 

Training schedule and attendance. Five whole-group training sessions and a separate 
training for new participants were held in year two. Optional supported work sessions for 
completion of the professional development products also were available. 

Training sessions. Training sessions were designed to provide participants with a greater 
understanding of science, technology, and engineering instruction and professional development, 
as well as the technical skills needed to create online professional development products. 
Training content was aligned with the stages that the teacher leaders had achieved in the 
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planning and creation of their professional development products. For example, it was decided 
that the May session would be dedicated for groups to solely work on their products. Participant 
feedback surveys helped program staff improve and modify trainings accordingly. 

Product development. The original STELP plan projected that the online products would 
be completed and launched at the end of year one. However, at the completion of the first year of 
STELP, the professional development products had not fully met the criteria for the program. 
The revised program plan specified that the products would be completed during year two of 
STELP, after teacher leaders received further training and additional time for the completion of 
their work. Seven of the eight products were launched and shown to science resource teachers at 
the beginning of September 2012. 

3. What was the impact of the STELP training sessions on teacher leaders? 

Teacher leaders’ perceptions of training sessions. Large percentages of participants 
responded with positive perceptions about the training. The statements they agreed with 
included: clear goals, objectives met, trainers knowledgeable and prepared, a comfortable 
environment, opportunities to reflect, questions were answered, and helpful information and 
skills gained. A majority also reported that the peer review feedback on their professional 
development product and the training plan were helpful; and that the article discussion on 
science standards was beneficial. Almost all agreed that the STE instructional specialist assigned 
to their group was helpful. The most dominant challenge reported by participants was time to get 
everything done followed by videotaping challenges. Nearly all participants agreed that the 
expectations were clear and that the expectations for next steps were clearly communicated. 

Teacher leaders’ perceptions about their technology skills. Participants’ comfort level 
with various Movie Maker technology skills greatly increased from the first session to the fifth 
session. Participants already had a fairly high comfort level with flip-style camera technology 
and PowerPoint. 

Teacher leaders’ understanding of science standards and practices. Participants’ self- 
measurement of their understanding of science standards and practices increased greatly from the 
first session to the fifth session. These included: articulate look-fors in evaluating student 
proficiencies, how K-8 technology standards are relevant to their STE teacher role, and how the 
practices of science and engineering and the framework for science education are relevant. 

Participant reflections. At the end of the school year, many participants gave specific 
examples of how the STELP professional development impacted their instruction including 
using discourse in their classroom and sharing information at school staff meetings. The 
opportunity to network and work with other MCPS professionals was most often mentioned as a 
benefit. 


Fined online professional development modules. The STELP participant teams turned in 
their final products in June 2012. A subgroup of participants served as editors for the final 
products. Seven of the eight product modules were launched and shown to science resource 
teachers at the beginning of September 2012. An evaluation protocol was developed, based on 
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the rubric created by STE staff, to use while viewing each of the online product modules to 
determine if they met the key expectations that were established. After viewing the seven 
products, all products were found to be highly proficient in the description of the practice, 
showing the impact on student learning, showing instructional strategies and using multimedia; 
only one product was found highly proficient with navigation. 

4. When and how were the online products made available to MCPS teachers? 

All the online product modules were combined into one product in a single location so that 
teachers may launch whichever product they wish to view using one main page. The product 
modules were initially shared with science resource teachers via a Webinar they participated in at 
the beginning of September 2012. 


Recommendations 

The following recommendations are suggested by the year two findings: 

• Resolve technical issues and logistical difficulties on existing product modules (e.g., the 
navigation and pace of slides, unclear font, inoperable website links, etc.). 

• Encourage wide-spread use of the modules among MCPS STE teachers and staff. 

• Collect data on who is viewing the modules, via the product modules. 

• Promote viewer feedback on the product modules and use the feedback to continue 
improving and expanding the product modules. 

• Continue providing time for team work, clear expectations, a time line, a training plan, 
and continued education on science practices and standards, for upcoming trainings as 
teacher leaders continue their work on developing and creating online product modules. 

• Monitor ways in which teacher leaders are sharing their STELP information with their 
school staff and ways in which they are improving their own, or their schools’, teaching 
of science technology and engineering. 


vii 


Program Evaluation Unit 


Evaluation of STELP, Year Two 



Montgomery County Public Schools 


Office of Shared Accountability 


This page is intentionally left blank. 


viii 


Program Evaluation Unit 


Evaluation of STELP, Year Two 



Montgomery County Public Schools 


Office of Shared Accountability 


Evaluation of the 

Science, Technology, and Engineering Leadership Program, 

Year Two 

Natalie L. Wolanin and Julie H. Wade 

Background 

The overarching vision for science, technology, and engineering (STE) instruction in 
Montgomery County Public Schools (MCPS) is that all students achieve full literacy in these 
areas. Students who are literate in science, technology, engineering, and mathematics (STEM) 
are knowledgeable, informed citizens who are able to think critically about concepts and solve 
problems. MCPS supports this vision by engaging all students through seamlessly integrated 
instruction that is project/problem and standards based (MCPS, 2012). 

The Science, Technology, and Engineering Leadership Program (STELP) was introduced in 
MCPS in 2010. The program aims to grow instructional capacity in MCPS by training and 
supporting a cadre of teacher leaders to design and deliver online professional development. 
Building on the skills and knowledge developed through the Elementary Science Leadership 
Program (ESLP), as well as tapping into the expertise of content specialists in secondary schools, 
STELP is preparing a group of teacher leaders to develop online materials to support inquiry- 
based instruction within effective, research-based teaching practices. With the creation of these 
resources, STELP aims to build a professional development network in science, technology, and 
engineering for wide use across MCPS. 

The ultimate goal for the three years of STELP is to improve instruction in science, technology, 
and engineering and, in turn, help students achieve STEM literacy (MCPS, 2010). This goal is in 
alignment with the mission of MCPS, “To provide a high-quality, world-class education that 
ensures success for every student through excellence in teaching and learning,” and with Goals 1 
and 2 of Our Call to Action: “Ensure success for every student,” and “Provide an effective 
instructional program” (MCPS, 2012). 

Research reported in Taking Science to Schools and Ready, Set SCIENCE: Putting Research to 
Work in K-8 Science Classrooms (Michaels, Shouse, & Schweingrube, 2007; National Research 
Council, 2007) is the basis for the National Research Council’s Framework for K-12 Science 
Education released July 2012. This framework, in turn, is the springboard for the development 
of the Next Generation of Science Standards (NGSS) managed by the nonprofit organization, 
Achieve, Inc. These standards are scheduled to be released in spring 2013. The strands of 
scientific proficiency represent learning goals for students and address the knowledge and 
reasoning skills that students must acquire to be considered fully proficient in science. They are 
also a means to that end: they are practices that students need to participate in and become fluent 
with in order to develop proficiency. 
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The year one evaluation plan projected that the trained teacher leaders would complete 
professional development products within the first year of the program, and that the products 
would be launched during the summer or fall of 2011. However, program administrators 
recognized the need for further skill development and time for teacher leaders to refine and 
complete their products. The program schedule was extended to allow more time for teacher 
leaders to refine and finalize their online products during year two. 

This report addresses the evaluation of the second year of STELP, focusing on the continued 
training of a cadre of teacher leaders to design and deliver online professional development 
product modules for other MCPS STE educators to view. This second year also provided an 
opportunity to upgrade the product modules, as the national documents evolved. The year two 
evaluation assesses the implementation of the STELP training protocol that was delivered to the 
participating teacher leaders. In addition, the online professional development product modules 
are reviewed and assessed according to a rubric reflecting effective online professional 
development in science, technology, and engineering. The objectives for the second year of 
STELP were — 

• continue the training and support of a group of teacher leaders by providing skills and 
knowledge to produce online professional development products that are based on a 
rubric reflecting characteristics of effective online professional development in science, 
technology, and engineering; and 

• publish the online professional development products created by STELP participants for 
use by MCPS teachers. 

The evaluation was requested by the Office of Curriculum and Instructional Programs (OCIP) 
and conducted by the Office of Shared Accountability (OSA). Funding was provided by a grant 
from the Howard Hughes Medical Institute (HHMI) to MCPS. 

Literature Review 

In a recent nationwide study, Wei, Darling-Hammond, and Adamson (2010) reported that 
teachers rated professional development in their subject area as their highest priority for further 
training. Consistent with this finding, teachers in an earlier study reported that professional 
development focusing on content knowledge was one of two elements that had the greatest effect 
on their knowledge and skills, and led to changes in instructional practice (Garet, Porter, 
Desimone, Birman, & Yoon, 2001). 

In challenging budgetary times, it has become increasingly important to make the most efficient 
and effective use of limited resources in all areas of education, and professional development is 
no exception. Dahlberg and Philippot (2008) conducted a study to explore the perceived needs 
and perceptions of teachers regarding their professional development. The researchers 
concluded that there is no one-size-fits-all model to meet the professional development needs of 
teachers, arguing that professional development should be differentiated according to the varying 
needs and career stages of teachers. They advocate for a collaborative approach to determining 
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professional development agendas, suggesting that, “Teachers, the ones who work most closely 
with the curricula and students, often know best where gaps in their own pedagogy and 
knowledge exist” (Dahlberg & Philippot, 2008, p. 22). 

As administrators have sought to stretch professional development dollars while providing 
teachers with accessible and meaningful professional development opportunities in their subject 
areas, interest in online professional development has grown (Dede, Ketelhut, Whitehouse, Breit, 
& McCloskey, 2009; National Research Council, 2007; Sawchuk, 2009). The flexibility of 
online professional development, as well as the capacity to tailor it to meet varying needs, makes 
it an attractive option in many school systems. As increasing numbers of teachers have 
participated in online professional development activities in recent years, evaluative research has 
not kept up with the growing use of these online models (Dede, et al., 2009). 

Dede and colleagues (2009) at the Harvard Graduate School of Education conducted a review of 
studies of online teacher professional development and noted that evidence of effectiveness was 
often lacking or anecdotal. In response to the scarcity of empirical findings, they developed a 
research agenda to help guide the study of online professional development toward a framework 
that would integrate theory and evidence-based practice. Among their recommendations are 
“research methodologies that do not simply replicate methods used in studying face-to-face 
professional development, but instead take advantage of the unique data collection possible in 
online programs” (Dede et al., 2009, p. 20). Their report also points out that since teachers apply 
what they learn over time, data should be collected over time as well. Consistent with the 
evaluation model constructed by Guskey (2000), Dede and his colleagues (2009) recognized the 
various levels of experience and learning to be addressed in an evaluation of professional 
development. They maintained that more and better measures implemented over time would 
help build understanding of what teachers learned in professional development, how they applied 
the new knowledge and skills to practice, and what changes resulted (Dede, et al., 2009). 
Consistent with the recommendations of Dede and colleagues (2009) in their “Research Agenda 
for Online Teacher Professional Development,” this evaluation includes data collected over time 
so that information about teachers’ use of the knowledge and skills gained from the professional 
development may be better understood. 

An evaluation of the first year of STELP was published in February 2012 (Wolanin & Wade, 
2012). The report assessed the year one implementation of the project through multiple surveys 
of participants, interviews with program administrators, and document reviews. Participants were 
positive in their perceptions of the training and reports about skills and knowledge they had 
learned. Feedback from the participants was used to develop recommendations for year two, 
including clarifying the STELP vision and understanding of STELP strands, collecting regular 
updates on the products’ progress, and providing opportunities for teams to work on their 
products. 
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Scope of the Study 

The evaluation addressed the second year of the Science, Technology, and Engineering 
Leadership Program. The focus of the year two evaluation was on implementation of the 
program in terms of continued training of a group of teacher leaders to develop online 
professional development modules for other MCPS science, technology, and engineering 
educators. Year two included both new and returning teacher leaders and built upon the online 
professional development modules created in year one. Toward this end, the evaluation 
addressed the following questions: 

1. What was the context of the Science, Technology, and Engineering Leadership Program? 

a. What were the characteristics of the teacher leaders who participated in STELP, 
including degree and experience? 

b. How many participated in year one? 

c. What were the characteristics of the schools with participating teacher leaders? 

2. To what extent was the training of teacher leaders implemented as planned? 

a. How was year two of the program organized and administered? 

b. Was the training schedule followed as planned? 

3. What was the impact of the STELP training sessions on teacher leaders? 

a. What were teacher leaders’ reactions to training, including the process for 
creating professional development products? 

b. What knowledge and skills did they gain? 

c. Did they have the resources and support needed to apply what they learned? 

d. How did they use the new information, to develop online products to meet the 
required criteria? 

4. When and how were the online products made available to MCPS teachers? 

a. What was the method of disseminating the products (e.g., website, SharePoint)? 

b. When and how were teachers informed of the products’ availability? 


Methodology 

Participation in STELP was limited to a group of teacher leaders selected by program staff, so a 
nonexperimental design utilizing a variety of data collection methods was applied. Data 
collection methods included the following: 

• Reviews of program documents and training records and materials, including 
professional development plan, session agendas, session handouts, session attendance 
records 

• Interviews with the STE project manager and instructional specialists 

• Surveys of teacher leader participants after each training session 

• Evaluation of online professional development modules 
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Study Sample 

In the second year of the evaluation, all teacher leaders enrolled in STELP comprised the study 
sample. A total of 47 school-based staff members participated during year two, representing 31 
elementary schools, 15 middle schools, and 1 high school in MCPS. Participants consisted of 
elementary, science, technology, engineering, and staff development teachers. 


Data Collection Activities 

To address the first evaluation question, “What was the context of STELP,” data were drawn 
from teacher leader participant feedback surveys, program records, and MCPS records to 
describe the participants and their schools. 

Assessment of the second evaluation question, “To what extent was the training of teacher 
leaders implemented as planned,” included a review of documents and interviews with program 
staff to determine the program training plan and schedule of training activities. 

To address the third evaluation question, “What was the impact of the STELP training sessions 
on teacher leaders,” Guskey’s (2000) model for evaluating professional development was used. 
Four of Guskey’s sequential levels were addressed in the second year of the evaluation: 
participants’ reactions, participants’ learning, organization support and change, and participants’ 
use of new knowledge and skills. Table 1 outlines the levels of Guskey’s model along with the 
evaluation activities that were used to address each level. 

Table 1 


Evaluation Activities Using Guskey’s Model for Evaluating Professional Development 


Level of evaluation 

Instrument/activity 

Data collected 

1. Participants’ reactions 

Surveys of participants 
(administered after each 
training) 

Participants’ satisfaction and reactions to 
professional development 

2. Participants’ learning 

Surveys of participants 
(administered before and after 
training) 

Participants' knowledge of effective science, 
technology, and engineering instruction; 
skills and knowledge required to plan and 
create online professional development 



resources 

3. Organization support and 
change 

Surveys of participants 
(administered after each 
training) 

Organizational support and teacher leader 
needs in the project 

4. Participants’ use of new 
knowledge and skills 

Surveys of participants 
(administered after each 
training); review of online 
products 

Participants’ reported use of new knowledge 
as they created professional development 
products; review of online products 
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Based on program goals and objectives and professional development materials and curricula, 
OS A evaluators collaborated with staff from OCIP to develop the evaluation instruments. The 
following instruments were developed during the second year of the evaluation: 

• Surveys of the teacher leader participants administered at the end of each group training 
session — the surveys assessed teacher leaders’ perceptions of the training received in the 
program (Appendix A). 

• Evaluation rubric for reviewing the end of year final professional development 
modules — OSA staff developed their evaluation rubric based on the rubric developed by 
OCIP staff (Appendix B). 

• End-of-year interview of STE program staff — the interviews assessed staffs’ perceptions 
of the trainings provided and the product development process by teacher leader teams 
(Appendix C). 


Summary of Data Analysis 

Procedures included a descriptive statistical analysis of teacher leaders’ survey data and a 
descriptive summary of the following: 

• Characteristics of participants 

• Attendance at professional development sessions 

• Participants’ reflections provided to STE program staff 

• STE program staff interviews 

• Professional development product module evaluations 
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Description of the STELP Year Two 

Invitation and Enrollment of Participants 

The cadre of teacher leaders who participated in year one were invited to return in year two. In 
addition, new teachers were recruited to expand the number of technology education and middle 
school teachers, and to replace teachers who did not return. New teacher leaders attended an 
introductory training session in September, prior to the regular schedule of training sessions. A 
total of 47 teachers signed on to participate in year two of STELP; 30 of those teachers were 
returning participants from year one. The teachers who did not return either had a change in 
position or position responsibilities. 


Training Sessions for Teacher Leaders 

In addition to the new participant training held in September, five professional development 
sessions were held in the second year: October 2011, December 2011, February 2012, 

April 2012, and May 2012. Attendance at the beginning was high for these sessions with 
approximately 38-40 of the 47 participants attending the first two sessions, and then 
approximately 31 of the total 47 participants attending each of the final sessions. 

In the new participant training session, teacher leaders learned about the vision for STE in 
MCPS, identified proficiencies in science and engineering, and learned technology skills needed 
for STELP, such as the use of video recording during classroom instruction and the use of 
Windows Movie Maker and Microsoft PowerPoint. 

In the first session, held in October 2011, teacher leaders were assigned their professional 
development team for the year, and they reviewed and provided feedback on the products 
developed in year one. Teacher leaders also learned about the charge of STELP, science and 
engineering proficiency, and how the strands for science and engineering proficiency and the 
MCPS STE vision aligns to the Framework for K-12 Science Education and the NGSS. In 
December, participants received camcorders, training plan templates, and task time lines. They 
also worked on their team projects; received tips for collecting good video and creating 
presentations; and listened to the guest speaker, Dr. Stephen Pruitt, Vice President of Achieve, 
Inc., talk about the framework, practices, and standards. In February, they continued to learn 
about the NGSS, as well as worked on their team projects and reviewed peers’ training plans. In 
April they received new camcorders (the initial camcorders had many audio difficulties), and 
worked on their team project and reviewed peers’ presentations. The final scheduled session in 
May was dedicated time for teams to work on their product modules. 
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Formation of Teams and Selection of Topics 

Eight teams, comprised of five to seven teachers, were formed at the first training session in 
October and were based on participants’ requests. The teams were each assigned one of the 
science and engineering framework practices, listed in Table 2, and were expected to develop an 
online STE professional development product about that practice. Each practice corresponds to 
one of four learning strands developed by the National Research Council (NRC) Committee on 
Science Learning. Some repeat participants worked on the same practice as year one and some 
worked on a new practice in year two. Also, each team was assigned an STE staff member as a 
support and “go to” resource person throughout the year. Each team self-assigned roles for their 
members such as: coordinator, check for understanding developer, quality controller, PowerPoint 
expert. Movie Maker expert, and any other roles they deemed useful. 


Table 2 

STELP Online Professional Development Design Teams and their Assigned Topics 


Science & Engineering 
Framework Practice # 

Framework Practice 
Name 

Target Proficiency 
Strand 

Practice 1 

Asking Questions and 
Defining Problems 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 2 

Developing and Using 
Models 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 3 

Planning and Carrying 
Out Investigations 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 4 

Analyzing and 
Interpreting Data 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 5 

Using Mathematics and 
Computational Thinking 

Strand 2: Generating and evaluating scientific 
evidence or technological solutions 

Practice 6 

Constructing 
Explanations and 
Designing Solutions 

Strand 1 : Know, use, and interpret scientific 
explanations of the natural and design world 

Practice 7 

Engaging in Argument 
from Evidence 

Strand 4: Participating productively in practices 
and discourse of science and engineering 

Practice 8 

Obtaining, Evaluating, 

Communicating 

Information 

Strand 4: Participating productively in practices 
and discourse of science and engineering 
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Product Development 

Eight teams were to produce eight online professional development product modules based on a 
rubric of elements that are characteristic of effective online professional development in science, 
technology, and engineering. Each product module represented one of the eight science 
practices. Products were a continuation of products created from year one. Some products used 
video clips and presentations from year one, and some were recreated using new video clips. 
Presentations were created using advanced features in Microsoft PowerPoint and Windows 
Movie Maker and then combined for a cohesive product using Articulate Studio ‘09, where 
viewers may select the professional development module they want to view. Teams created 
training plans to be used for planning their professional development product. The draft training 
plans were presented for review by peers and an STE specialist (Appendix D). The training plan 
was a “5E” (engage, explain, explore, evaluate, and extend/elaborate) training plan and included 
the following criteria: 

• Varied assessment strategies before, during, and after professional development 

• Variety of opportunities for viewer of the professional development to engage with, 
explore, process, and reflect on content 

• Accurate and adequate content about session topic (specific science and engineering 
practice) 


Findings 

1. What was the context of the Science, Technology, and Engineering Leadership 
Program? 

Characteristics of the Participating Teacher Leaders 

A total of 47 teachers (31 elementary school, 15 middle school, and 1 high school) participated 
in the second year of STELP; although 4 participants withdrew midyear because of personal or 
professional reasons. Of the 47 teachers, 30 participated in the first year of STELP as well. 

The participants represented a variety of positions in both elementary and secondary schools. Of 
the 47 who participated, 28% taught Grades 3 through 5, and 21% taught middle school science. 
Almost everyone had more than five years teaching experience (98%), and the majority reported 
having experience teaching science (92%). Three of the middle school teachers also were 
science resource teachers, and two of the middle school teachers also were technology resource 
teachers. Characteristics of those who agreed to participate in STELP are summarized in 
Table 3a. 
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Table 3 a 


Science, Technology, and Engineering Leadership Program: 
Characteristics of Participating School Staff 


Total 

Teacher Leaders 
(IV = 47) 

Elementary 
Teacher Leaders 
(77= 31) 

Secondary 
Teacher Leaders 
(N= 16) b 

Current position 

n 

% 

n 

% 

n 

% 

3-5 teacher 

13 

27.7 

13 

41.9 

0 

0.0 

MS science teacher 2 

10 

21.3 

1 

3.2 

9 

56.3 

Staff development teacher 

7 

14.9 

6 

19.4 

1 

6.3 

K-2 teacher 

7 

14.9 

7 

19.4 

0 

0.0 

Tech and engineering teacher 

5 

10.6 

0 

0.0 

5 

31.3 

Special education teacher 

2 

4.3 

2 

6.5 

0 

0.0 

HS science teacher 

1 

2.1 

0 

0.0 

1 

6.3 

ES science teacher (multi grades) 

1 

2.1 

1 

3.2 

0 

0.0 

Focus teacher 

1 

2.1 

1 

3.2 

0 

0.0 

Total years teaching 

1^1 years 

1 

2.1 

1 

3.2 

0 

0.0 

5-15 years 

34 

72.3 

21 

67.7 

13 

81.3 

16+ years 

12 

25.5 

9 

29.0 

3 

18.8 

Years teaching science 

None 

4 

8.5 

0 

0.0 

4 

25.0 

1^1 years 

34 

72.3 

22 

71.0 

12 

75.0 

5-15 years 

0 

0.0 

0 

0.0 

0 

0.0 

16+ years 

9 

19.1 

9 

29.0 

0 

0.0 

Years teaching technology 

None 

31 

66.0 

22 

71.0 

9 

56.3 

1^1 years 

5 

10.6 

3 

9.7 

2 

12.5 

5-15 years 

6 

12.8 

4 

12.9 

2 

12.5 

16+ years 

5 

10.6 

2 

6.5 

3 

18.8 

Years teaching engineering 

None 

34 

72.3 

23 

74.2 

11 

68.8 

1-4 years 

7 

14.9 

4 

12.9 

3 

18.8 

5-15 years 

4 

8.5 

3 

9.7 

1 

6.3 

16+ years 

2 

4.3 

1 

3.2 

1 

6.3 

Degree or certification in science 

, technology, or engineering 





No 

29 

61.7 

27 

87.1 

2 

12.5 

Yes 

18 

38.3 

4 

12.9 

14 

87.5 


“Three are also middle school science resource teachers and two are technology resource teachers. One middle school 
science teacher is from a K-6 elementary school. 

b One secondary teacher leader was at the high school level; all others were at the middle school level. 


Nearly two thirds (64%) of year two participants also participated in year one, with almost three 
fourths (74%) of the elementary school teacher leaders having participated in year one of STELP 
(Table 3b). Forty percent of participants reported holding a current leadership role at their school 
such as team leader, resource teacher, or leadership team, and an additional 23% reported having 
some other leadership or professional development role such as a specified school program 
coordinator or leader or a Gifted and Talented (GT) liaison for the school. 
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Table 3b 


Science, Technology, and Engineering Leadership Program: Previous Training and Leadership 
Experience of Participating Elementary and Middle School Staff 


Total 

Teacher Leaders 
(A = 47) 

Elementary Middle School 

Teacher Leaders Teacher Leaders 

(A =31) (A =16) 

n % 

n % \ n % 

Participation in year 1 STELP 

leadership/PD role 30 63.8 

23 74.2 7 43.8 

Team leader, RT, SDT leadership team 19 40.4 

13 41.9 6 37.5 

Other school leader (i.e. PBIS/green school 
leader, GT liason, content specialist, subject 

coordinator) 1 1 23.4 

8 25.8 3 18.8 

Note. PD = professional development; SDT = staff development teacher; PBIS = positive behavioral interventions and 


supports; GT = gifted and talented. 


Experience with online training. Of the 17 teacher leaders who were new to STELP this 
year, 5 reported having taken an online course, and 3 reported having participated in a Webinar 
as their participation in any kind of online training; the remaining did not respond. 

Characteristics of Schools with STELP Teachers 

Teachers who participated in STELP were from 22 elementary schools, 15 middle schools, and 
1 high school. Characteristics of the schools represented are shown in Table 4. 

On average, as a group, schools with teachers participating in STELP had proportions similar to 
MCPS averages in terms of students receiving Free and Reduced-price Meals System (FARMS) 
services (35% for STELP elementary schools and 37% for MCPS) and students receiving 
English for Speakers of Other Languages (ESOL) services (21% of STELP elementary schools 
and 21% for MCPS; 4% for STELP middle schools and for 5% for MCPS). However, STELP 
schools at the middle school level had a lower proportion of FARMS recipients than MCPS 
(23% compared to 33%). Just as there is a wide range of proportions of FARM recipients in 
MCPS schools, there is a wide range among schools with teachers participating in STELP (2- 
74% for elementary school; 1-56% for middle school). Additionally, the range of ESOL 
recipients in elementary schools with participating STELP teachers is 2-60%, and 0-9% for the 
middle schools. 


Table 4 


Science, Technology, and Engineering Leadership Program: 
Characteristics of Elementary and Middle School Participants 


School-level 

characteristics 


Elementary 

STELP 

(22 schools) MCPS 

Middle 

STELP a 
(15 schools) 

MCPS 

Number of students 

Mean 

Range 

539 

261-1036 

531 

94-1036 

862 

580-1338 

815 

380-1338 

% of students eligible for 

Mean 

34.9 

36.6 

23.0 

32.7 

FARMS 

Range 

2.2-73.8 

1.0-94.6 

1.4-55.5 

1.4-61.5 

% of students enrolled in 

Mean 

20.6 

21.3 

4.3 

4.6 

ESOL classes 

Range 

2.1-59.6 

0.0-72.9 

0. 1-8.5 

0.0-12.1 


Note. Based on 2012 MCPS data. 

‘‘Middle school only is shown; table does not include one participant from high school so as to not skew findings. 
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2. To what extent was the training of teacher leaders implemented as planned? 

Training Schedule and Attendance 

The program plan specified that a new participant training would be held in September. 
Additionally, whole-group training sessions would be held in October, December, February, 
April, and May. Optional supported work sessions for completion of the professional 
development products would be offered in May and June. All of these sessions were held as 
planned; attendance is shown in Appendix E. 

Precise attendance counts for each session are unknown. The attendance counts are based on the 
number of feedback surveys returned; therefore, attendance may be underestimated. These 
counts show that attendance at each of the training sessions was fairly high with a lower 
attendance for sessions held later in the school year. A range of 31-40 teacher leaders attended 
the sessions from September through April. Attendance at the May session and the end of the 
year optional work sessions was not recorded. 

Training Sessions 

The purpose of the training sessions was to provide teacher leader participants with a greater 
understanding of science, technology, and engineering instruction and professional development 
as well as the technical skills needed to create online professional development products. The 
stated learning outcomes for each of the training sessions are shown in Appendix E. The training 
session agendas were planned to align the training content with the stages that the teacher leaders 
had achieved in the planning and creation of their professional development products. For 
example, it was decided that the May session would be dedicated for groups solely to work on 
their products. Also, findings from the participant feedback surveys helped program staff 
improve and modify trainings accordingly. For example, in response to participants’ feedback, 
the STE staff stated at the December session that they would schedule large blocks of time in all 
of the sessions for teams to work on their products, and they would focus on providing clear 
outcomes and time lines. The session feedback surveys were summarized by the evaluators and 
provided to program administrators after each training session. 

Product Development 

The original STELP plan projected that the online products (also called modules) would be 
completed and launched at the end of year one. However, at the completion of the first year of 
STELP, the professional development products were determined not to have fully met the criteria 
for the program. Technical difficulties, time, challenges collecting quality video clips, and lack 
of a clear understanding of the expectations for the finished product were the primary issues that 
prevented the launch of products at the end of year one. Program managers determined that more 
time and training were needed by the teacher leaders to create high-quality professional 
development products. Thus, the time frame for the program was adjusted. The revised program 
plan specified that the professional development products would be completed during year two 
of STELP, after teacher leaders received further training and additional time for the completion 
of their work. 
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In year two, not only were the products revised to resolve technical issues and present a higher 
quality of professionalism, all of the modules were upgraded as the project evolved in 
accordance with developments in national standards. In year one, the modules focused on the 
four proficiency strands as outlined in Ready, Set, Science and the K-12 Framework for Science 
Education. The products in year two provided further details under the original four strands, and 
were aligned with the NGSS. 

The leadership teams turned in their final product modules at the end of the school year in 
June 2012. STE program staff, along with a team of chosen STELP participants, then worked 
through the summer on fine-tuning the products and resolving technical issues. They also 
formatted the product modules, using the program Articulate Studio ‘09, and created one 
cohesive location where teacher participants would be able to access and view any of the 
modules. Seven of the eight products were launched and shown to science resource teachers at 
the beginning of September 2012. 


3. What was the impact of the STELP training sessions on teacher leaders? 

Teacher Leaders’ Perceptions of Training Sessions 

In surveys administered after the October and December training session, teacher leaders were 
asked to indicate their level of agreement with questions about the training session. Teacher 
leaders’ perceptions of the trainings are summarized in Table 5. 

Across the two sessions, large percentages of teacher leader participants responded with positive 
perceptions about the training, with 100% agreeing with most of the statements. The statements 
included: clear goals, objectives met, trainers knowledgeable and prepared, a comfortable 
environment, opportunities to reflect, questions were answered, and helpful information and 
skills gained. 


Table 5 

Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Three Sessions 


% Respondents Strongly 
Agree/Agree 


October 

December 

2011 

2011 

II 

(N =38) 


The goals of today’s training were clear. 

97.3 

100.0 

The objectives of today’s training were met. a 

100.0 

92.1 

The trainers were knowledgeable and well prepared. 

100.0 

100.0 

An environment was created in which I felt comfortable taking risks. 

100.0 

100.0 

Opportunities were provided for me to process and reflect upon the application of the 
knowledge and skills learned. 15 

100.0 

94.6 

My questions dining the training today were answered adequately. 15 

100.0 

97.4 

As a result of today’s training, I have gained information and skills that will help me in 
this role. 

100.0 

97.3 


Note : Includes staff who attended the October session, but later dropped out of the program. 
“October N = 35. 

b October N = 36, December N = 37. 
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At the February training session, all participants (100%) reported that the training plan was very 
or somewhat helpful; almost all (95%) participants reported that the peer review feedback on 
their professional development product was very or somewhat helpful (Table 6). More than four 
fifths of participants (83%) reported that the article discussion on science standards was very or 
somewhat helpful. Finally, almost all (97%) of participants at the April training session said that 
the STE instructional specialist assigned to their group was very or somewhat helpful. 


Table 6 

Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Three Sessions 


% Responding 
Very/Somewhat Helpful 


How helpful was. . . . 

February 
2012 
(N = 40) 

April 2012 
(N = 30) 

the training plan in developing your online PD? 

100.0 

N/A 

the feedback provided by the peer review in developing your online PD. a 

95.0 

N/A 

today's article discussion on Science standards in further understanding the goals of 
STELP? 

82.5 

N/A 

the STE Instructional Specialist assigned to your group? 

N/A 

96.7 


Note. PD = professional development; N/A = not applicable (did not administer question). 

d The agenda was modified during the April administration not to include peer review so that participants could have a longer 
time to work on their products so this question is not applicable in April. 


In order to help plan for upcoming training sessions, teachers were asked at the February training 
session how useful they would find training on certain technical skills. Almost three fourths 
(73%) of the participants said that learning about editing and storing on Movie Maker would be 
very or somewhat useful (Table 7). Using the flip-style camera was found to be the least useful, 
with under one third (32%) rating it very or somewhat useful. 

Table 7 


Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Three Sessions 


How useful would you find training for ... 

% Responding 
Very/Somewhat Helpful 
February 2012 
(A =37) 

editing using Movie Maker. 

73.0 

storing using Movie Maker. 

73.0 

retrieving video clips using Movie Maker. 

60.5 

using Microsoft PowerPoint for online PD products. 

56.8 

using flip-style camera technology. 

32.4 
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Summary of Teacher Leaders Responses to Open-ended Questions 

In each of the training session surveys, teacher leader participants were asked a variety of open- 
ended questions about STELP. Additionally, after the last session was completed, participants 
were sent a reflective survey by the STE program staff. Teacher leaders’ responses to the open- 
ended survey questions and the program staffs open-ended reflective questions are summarized 
below. 


Important aspects of training. At the end of four training sessions, teacher leader 
participants were asked, in an open-ended question, to identify the most important thing gained 
from the trainings. The top mentions for each training session are shown below. 

Most important from October training (N = 34) 

• Learning about the science practices and strands (38%) 

• Having a clearer direction and understanding of the project (26%) 

Most important from December training ( N = 40) 

• Guest speaker Dr. Stephen Pruitt (27%) 

• Receiving a time line (19%) 

• W orking with their te am (16%) 

Most important from February training (N = 40) 

• Gaining a clear vision and clear expectations (62%) 

• Having a training plan (35%) 

• Working with their team (33%) 

Most important from April training (. N = 29) 

• Working with their team (48%) 

• Working on their PowerPoint (31%). 

Suggestions for improving the training. In four of the five training sessions, participants 
were asked for any suggestions they had to improve the trainings. The number of participants, 
during each session, who responded to this question were: 18 in October, 23 in December, 19 in 
February, and 17 in April. Many of those responding to this question submitted individual 
suggestions (50% in December and 68% in February) that were not suggested by others. 
However, more time was suggested throughout the trainings, with a larger amount suggesting it 
in October (39%) and April (41%). Other suggestions made throughout the year were: more 
clarity and specifics (28% in October), more breaks and resources (22% in December), and a 
better camera and peer feedback (26% in February). Two participants made a final comment in 
May that they felt a couple days of training in a row might have been helpful. 

Challenges. In four of the five training session surveys, teacher leader participants were 
asked, “Are there any other challenges you are facing in your teacher leader role as you work 
with your team and develop your online product?” Time was the dominant area of concern. Of 
the 21 teacher leaders naming a challenge in the first survey, 48% reported that there was “not 
enough time to get everything done,” and “time is always a challenge.” These time concerns 
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were reported by 39% of the 23 teacher leaders who stated challenges in the second survey, 35% 
of the 26 in the third survey, and 27% of the 26 in the fourth survey. Some examples that the 
participants cited were: “just balancing and finding time;” “there are so many priorities at school 
that I often wish for more time to devote to this;” “uninterrupted time to work with my team is 
difficult to arrange;” and “our team worked diligently today, but did not progress to the point of 
having a solid training plan. It will now have to be done outside of training, which is difficult 
since time is already at a premium.” 

The other dominant area of concern across the training sessions was videotaping. At the 
beginning, there was more concern about finding the time to collect videos, and more comments 
were related to collecting quality videos. Almost one third of the teacher leader participants 
responding to each of the surveys, named videotaping as a challenge: 33% of 21 in October; 35% 
of 23 in December; 31% of 26 in February; and 27% of 26 in April. Some examples of the 
participants’ comments about the challenges of videotaping were “collecting a variety of artifacts 
from ALL grades;” “Worried about recording quality (i.e., movement, noise, lighting);” “Just 
figuring out how to know when to video and take pictures and when not to;” and “I’m concerned 
about collecting artifacts when there are so few opportunities to do so.” Some other examples of 
challenges mentioned were: technical difficulties, getting everyone in the team together, team 
agreement, and understanding the science practices or product. 

Other comments. In the surveys, teacher leaders were asked to elaborate on any of the 
statements they rated. Additionally, some teachers provided other undirected comments 
throughout the survey. Themes of comments pertaining to the trainings could be extracted: 

• The October trainings’ comments included: needing more clarity about the final product 
or next step; they gained helpful information on the science strands and standards; and 
appreciated time to revisit past projects and process information. There also were a few 
participants who thought the training was more organized and clearer compared to last 
year. 

• The December training’s comments included: that teacher leaders ran out of time to 
accomplish their tasks; they enjoyed the new time line and checklist provided; and the 
expectations and group direction was clearer. 

• The February training’s comments included: enjoyed the time given to work with their 
group, found the training plan helpful, said their team had more of a focused direction, 
and found the assigned article very informative and interesting, but perhaps not related 
directly to the task at hand. Additionally, many found the peer feedback helpful, although 
a couple of respondents commented that it was awkward or that they would have 
preferred more mixed up groupings for feedback. 

• The April training’s comments were mixed between feeling that their team made a lot of 
progress and feeling like they ran out of time and were not as productive as they felt they 
should be. 
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• The May training’s comments included praises for a great professional development 
experience 

Progress reported during training. At the first two training sessions, participants were 
asked about the clarity of expectations for the professional development products. Just about 
everyone (95% in October and 100% in December) agreed that the expectations were clear 
(Table 8). With the exception of a few participants in the first session, everyone in all the 
sessions agreed that the expectations for next steps were clearly communicated. For the last two 
to three training sessions, all or nearly all, participants agreed that: they felt good about their 
team’s progress; they felt good about the direction of their final product; expectations for what 
their team was to accomplish were clear; and that the training session helped their team’s 
progress. 


Table 8 


Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Three Sessions 




% Respondent Strongly Agree/Agree 



October 
2011 
(A =37) 

December 
2011 
(A =38) 

February 
2012 
(A =38) 

April 
2012 
(A =31) 

May 
2012 
(A =31) 

The expectations for this professional 
development product are clear. 2 

94.6 

100.0 

N/A 

N/A 

N/A 

The expectations for next steps were clearly 
communicated. 

91.9 

100.0 

100.0 

100.0 

100.0 

I feel good about the progress my teacher team 
has made so far. 

N/A 

N/A 

100.0 

87.1 

96.8 

I feel good about the direction my team’ s final 
product is headed in. 

N/A 

N/A 

100.0 

90.3 

96.8 

The expectations for what our teacher team is 
to accomplish for this project is clear. 

N/A 

N/A 

100.0 

100.0 

N/A 

I believe today’s session will help/has helped 
with our teacher teams’ progress as we develop 
our professional development topic and 
product. 

N/A 

N/A 

100.0 

93.5 

N/A 


Note. N/A designates not available because not asked. 
“December N = 35. 


Teacher leader participants also were asked in the last three training sessions about their 
perception of their team’s progress; their responses are reported in Table 9. In February, the 
majority (77%) felt they were ahead or right on schedule; in April the participants were split 
between being right on schedule (48%) and feeling they were behind schedule (58%). Finally, in 
the last structured training session in May, more than half of the participants felt they were ahead 
or right on schedule with over one third (37%) feeling they were behind. 
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Table 9 


Science, Technology, and Engineering Leadership Program: 
Teacher Leaders’ Perceptions of Training Provided in Three Sessions 


How would you describe your team’s progress on your PD 
product? 

February 
(N = 35) 

April 
(A- 31) 

May 
(A =30) 

Ahead of our planned time line 

5.7 

0.0 

13.3 

Right on schedule 

71.4 

41.9 

50.0 

Running behind our planned time line 

22.9 

58.1 

36.7 


Teachers Leaders’ Perceptions of Knowledge and Skills Gained 

Technology skills. In surveys administered at the first and fifth training session, teacher 
leader participants were asked to indicate their level of agreement with questions about their 
specific technology skills. Table 10 shows that teacher leaders’ comfort level with various 
technology skills greatly increased from the first session in October to the fifth session in May. 
Only responses from those participants’ who rated their skill in both sessions are shown. 
Approximately one half agreed or strongly agreed that they were skilled with processing, editing, 
storing, and retrieving video clips in Movie Maker (46%-59%) at the beginning of the training 
sessions. By the end of the training sessions, each of these percentages increased to more than 
four fifths (82-88%). Additionally, although a high percentage of respondents agreed or 
strongly agreed that they were skilled in flip-style camera technology (80%) and Microsoft 
PowerPoint (84%) at the first training session, these percentages increased to 96% and 100% 
respectively at the last training session. 


Table 10 


Science, Technology, and Engineering Leadership Program: 
Teachers’ Perceptions of Technology Skills in Two Sessions 



% Responding 
Strongly Agree/Agree 

Survey items 

A 

October 

2011 

May 

2012 

I am skilled in Flip style camera technology for capturing student 
learning. 

25 

80.0 

96.0 

I am skilled at using Windows Movie Maker software for. . . 

processing. 

23 

52.2 

87.0 

editing. 

24 

45.8 

87.5 

storing. 

22 

54.5 

81.8 

retrieving video clips. 

22 

59.1 

81.8 

I am skilled at Microsoft PowerPoint. 

25 

84.0 

100.0 


Note. N represents those participants who responded in both October and May sessions. 


An additional resource for gauging participants’ growth in technology skill level was used. At 
the end of the school year, the STE program staff sent out a reflection survey to all year two 
STELP participants (see Appendix F) and they conducted the data summary results. They asked 
the teacher leader participants, “Prior to participating in STELP, what was your skill level of 
Movie Maker?” Three fourths of the 29 responding participants gave a rating of “basic,” 
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17% “proficient,” and 7% “advanced.” Then participants were asked, “As a result of 
participating in STELP, what is your skill level of Movie Maker?” Of the same responding 
participants, only 14% gave a rating of “basic,” 55% “proficient,” and 31% “advanced.” 

The same two questions were asked about PowerPoint. The first question asked was, “Prior to 
participating in STELP, what was your skill level of PowerPoint?” Just about one fourth (24%) 
responded “basic,” over half (52%) gave a rating of “proficient,” and 24% “advanced.” When 
asked, “As a result of participating in STELP, what is your skill level of PowerPoint,” no one 
gave themselves a “basic” rating; half indicated “proficient,” and the other half “advanced.” 

Understanding of science standards and practices. Participants’ rated their 
understanding of science standards and practices at the first training session in October, again at 
the third training session in February, and at the final training session in May. At the beginning 
of the year, 91% of the teacher leaders agreed or strongly agreed that they could: articulate look- 
fors in evaluating student proficiencies; nearly three fourths (74%) reported they could explain 
how K-8 technology standards are relevant to their STE teacher role; and 87% could explain 
how the practices of science and engineering and the framework for science education are 
relevant to their STE teacher role (Table 11). At the final session in May, the percentage of 
teacher leaders who agreed or strongly agreed with these statements was 100% or almost 100%. 

Table 1 1 


Science, Technology, and Engineering Leadership Program: 
Teachers’ Understanding of Science Standards and Practices 


% Responding 
Strongly Agree/Agree 

Survey items 

N 

October 

2011 

February 

2012 

May 

2012 

I can articulate look-fors in evaluating student proficiencies in the 
practices of science and engineering. 

23 

91.3 

100.0 

100.0 

I can explain how K-8 technology standards are relevant to my 
role as a teacher and as a STE teacher leader. 

23 

73.9 

97.5 

100.0 

I can explain how the practices of science and engineering and 
the Framework for K-12 Science Education are relevant to my 
role as a teacher and as a STE teacher leader. 

23 

87.0 

100.0 

95.7 


Note: N represents those participants who responded in all three sessions: October, February, and May. 


Reflections 

Participant reflections. At the end of the school year, the STE program staff sent teacher 
leader participants a reflective questionnaire (see Appendix E.) It is important to note that this 
questionnaire was not collected anonymously, and narrative responses were submitted directly to 
the STE program staff supervisor. A summary of the information collected, relevant to the 
evaluation, follows. 

When teacher leader participants were asked to describe how the STELP professional 
development has impacted their instruction, 7 of the 28 respondents mentioned they are using 
discourse in their classrooms, and 12 gave other specific examples of implementations in their 
classroom (i.e., using investigations more often, using models, analyzing data, reflecting in 
journals, etc.). Five said they are incorporating or trying to incorporate the practices more into 
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their teaching (without giving specific examples), and three said they are incorporating more 
science and engineering into their teaching. Five said they have a better understanding of the 
NGSS and where science education is headed. Three participants said they are working with 
other teachers at the school to better improve how science is implemented in their class. 

Teacher leader participants also were asked to describe how the STELP professional 
development has impacted their work with colleagues or events at their school. Of the 29 
participants who responded, 23 said they shared information they have acquired — 17 shared it 
with other school staff members through various meetings; 4 shared it with either a colleague or 
the staff development teacher at their school; and 2 shared information with parents. Five 
participants gave examples of how they changed instruction, and five stated how they changed 
their schools’ annual inquiry conference (for example, changed it to a STEM-focused 
conference). Three stated they are considered a science resource or “go to” person at their 
school; and three participants cited other activities they implemented at their school such as 
clubs, a summer family STEM activity calendar, student inquiry challenges, and instruction on 
well-designed investigations to be implemented before the annual school STEM conference. 

The reflective questionnaire also asked participants if they would like to continue in the STELP 
program next year (i.e., if given the opportunity), and 23 of the 29 said they would like to 
continue. Three said they weren’t sure and three said “no,” citing work or personal conflicts. 

When asked to provide any additional comments on how they have benefited as a result of their 
participation in the STELP professional learning community, the aspect mentioned most often 
(16 of 29) was the opportunity to network and work with other MCPS professionals. Many 
stated that they now have a group of colleagues they may go to and bounce ideas off of. 

Instructional specialist reflections. The six STE instructional specialists were 
interviewed individually by OSA staff using a semi-structured questionnaire. All six specialists 
were involved in year one as well as year two; however, two of them had a more active role in 
year two. Each specialist was assigned one or two professional development design teams. The 
specialists described their role as being one of support and advisor to the team. The specialists 
met with their team during the scheduled participant trainings and were in touch with team 
members via e-mail. Some of the specialists met with their team separately, at the end of year. 
Although all specialists provided their assigned team with feedback and technical support, the 
depth of involvement by the specialist depended on the team and their needs. In some cases, 
involvement was minimal because, as one specialist described, the team was a “high functioning 
group.” In other cases, involvement was greater with the specialists giving much more feedback 
which may have involved changes to the team plan or product or refocusing of direction. In some 
cases, the specialist needed to help some new members catch up on technical training or 
understanding the practice. Also, some specialists took on a few tasks such as making 
adjustments to the product presentation or finding examples to use. It was reported that most 
team members worked well together with minimal conflict; however, there were some cases 
where some team members didn’t contribute or participate as much as others, or there was not 
someone who took a leadership role in the group. 
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All the specialists reported that the professional development and product development in year 
two was much better than year one. Reasons given for this positive turnaround were: much more 
built-in time for teams to work on their products; much clearer expectations for the end products; 
teams developed a training plan; teams built on last year’s products and knowledge of the 
concepts and technology; and finally, more established framework and practices. However, 
technical issues were still reported as the biggest challenge, mostly relating to the software 
Articulate Studio ‘09 (i.e., use, availability, and obtaining support); as well as video problems, 
consistency with versions of software, such as Movie Maker, and other unforeseen technical 
issues that arose. Overall, most of the specialists were happy with the final product their team 
turned in (before being completely finalized); however there were two cases where the 
specialists felt the products were not at the appropriate professional level. Some of the items that 
individual specialists mentioned wanting to see implemented next year were: continue offering a 
lot of work time; incorporate cross-cutting concepts; bring in high school level examples and 
participants; more peer reviews; availability of Articulate Studio ‘09 at the technology lab; and 
continue to build this professional learning community. Three specialists also mentioned that the 
national standards and framework aren’t final yet, so that would need to be considered as they 
move forward. Four of the six instructional specialists who were asked what they thought the 
most important thing teachers would get out of viewing the online professional development 
modules, and the responses were: learning about the new science standards; building leaders, 
learning about and working with other school levels besides their own, and understanding what it 
means to be proficient in engineering and science. 

Final Online Professional Development Modules 

The teacher leader teams turned in their final products at the end of the school year in June 2012. 
A subgroup of STELP participants was chosen to serve as editors for the final products. The 
editors worked on the product module in the summer and were charged with ensuring that all the 
projects met the expectations as outlined in the project rubric. They also were responsible for 
any additional movie editing, PowerPoint development, and script writing, before publishing the 
online product module to Articulate Studio ‘09. Seven of the eight product modules were 
launched and shown to science resource teachers at the beginning of September 2012. 

OSA staff developed an evaluation protocol, based on a rubric created by STE staff, to use while 
viewing each of the online product modules to determine if they met the key expectations that 
were established. Seventeen aspects were given a rating of “highly proficient,” “proficient,” or 
“not proficient.” Table 12 presents a summary of these findings (see Appendix B for the full 
evaluation protocol). 
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Table 12 

Evaluation of STELP Online Professional Development Product Modules 
(N = 7 reviewed products) 



Highly 


Not 

Not 

Total 


Proficient 

Proficient 

Proficient 

Available 

Products 


00 

00 

00 

00 

N 

Organization 

Sequence 

5 

2 



7 

Description of practice 

7 




7 

Use by scientists and engineering 

3 

3 

1 


7 

Impact on student learning 

7 




7 

Classroom exemplars 

6 



1 

7 

Instructional strategies 

7 




7 

Design 

Multimedia 

7 




7 

Navigation 

1 

6 



7 

Technical issues 

2 

4 

1 


7 

Visual presentation 

2 

5 



7 

Audio presentation 

2 

4 


1 

7 

Artifacts 

6 



1 

7 

Check for Understanding (CFU) 

Opportunities for CFU and 
reflection 

6 

1 



7 

Connection between CFU and 
presentation 

6 

1 



7 

Answer review for CFU 

4 

2 


1 

7 

Preassessment connection with 
presentation 

2 

3 


2 

7 

Goals/outcomes of presentation 
met 

6 



1 

7 

Cited Resources 

Resource for practice is cited 

6 modules cited a reference for practice 


7 

Cited research (not including 
practices) 

2 modules cited additional research 


7 


Among the areas which weren’t found to be highly proficient, many of the shortcomings were 
related to presentation, navigation, or technical aspects. Examples were: it was not always clear 
how to continue to the next slide or how to navigate throughout the module; sometimes the user 
was navigated incorrectly; slides were set to advance at a very fast pace or very slow pace; some 
audio was hard to hear or not in alignment; font was fuzzy or too small; and links to websites or 
videos did not work. 

Some examples which contributed to highly proficient ratings of some modules were: use of 
quotes, captions on video clips to further explain a strategy, various examples by grade or school 
level, clear directions on how to navigate throughout the module, examples used by scientists 
and engineers, many opportunities to check for understanding, and lots of resources available. 


Program Evaluation Unit 


22 


Evaluation of STELP, Year Two 













Montgomery County Public Schools 


Office of Shared Accountability 


4. When and how were the online products made available to MCPS teachers? 

The seven online product modules were combined into one product in a single location so that 
teachers are able to launch whichever product they wish to view using one main page. The 
product modules were initially shared with science resource teachers via a Webinar they 
participated in at the beginning of September 2012. 

Discussion and Recommendations 

This report addresses the second year of the Science, Technology, and Engineering Leadership 
Program. The focus of the year two evaluation was on implementation of the program in terms 
of continuing to train a group of teacher leaders to develop online professional development 
modules for other MCPS science, technology, and engineering educators to view. 

The 47 teacher leaders who participated in the second year of STELP were consistently positive 
in their perceptions of the training and support provided by the program. In surveys and 
interviews, teachers and STE instructional specialists were highly positive in their feedback 
about being given time during trainings to work on their team projects, the improvement in 
clarity and organization compared to last year, receiving a time line, and developing a training 
plan. Participants also reported great improvement in their technology skills related to 
developing the online products. Although they appreciated the time allotted to team work, they 
identified time, followed by technology difficulties as the greatest challenges. Participants also 
reported great improvement regarding their understanding of science standards and practices; 
however, there is still room for improvement in mastering these concepts, and participants 
mentioned learning about practices and strands as an important piece of information covered in 
trainings. In response to several of the surveys, the teacher leaders were split about whether they 
felt their team was on schedule with their product development or running behind their time line. 

Finally, at the completion of the second year of STELP, evaluation of the product modules by 
OSA researchers determined that most of the modules were proficient or highly proficient in 
meeting many of the expectations. However, the products also had room for improvement, 
primarily in the technical and logistical aspects of the presentations. 
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The following recommendations are suggested by year two findings: 

• Resolve technical issues and logistical difficulties on existing product modules (e.g., the 
navigation and pace of slides, unclear font, inoperable website links, etc.). 

• Encourage wide-spread use of the modules among MCPS STE teachers and staff. 

• Collect data via the product modules, on who is viewing the modules. 

• Promote viewer feedback on the product modules, and use the feedback to continue 
improving and expanding the product modules. 

• Continue providing time for team work, clear expectations, a time line, a training plan, 
and continued education on science practices and standards for upcoming trainings as 
teacher leaders continue their work on developing and creating online product modules. 

• Monitor ways in which teacher leaders are sharing their STELP information with their 
school staff and ways in which they are improving their own, or their schools’, teaching 
of science, technology, and engineering. 
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Appendix A: Training Sessions’ Feedback Surveys 

Survey Part I 

Science Technology Engineering Leadership Program (STELP) 

Year 2, Oct 27, 2011 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. This is the first in a series of surveys that will be given after each 
training session during year two of the program, thus our need to collect names; however, your 
answers are strictly confidential (i.e. answers will not be linked to individual names when 
reporting). 

Name: 

School: 


Please indicate how much you agree or disagree with the following items by checking the 


appropriate box. 

Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

1. I am skilled in using “Flip” style 
camera technology for capturing 
student learning. 

□ 

□ 

□ 

□ 

2. I am skilled at using Windows 
Movie Maker software for. . . 

a. processing 

□ 

□ 

□ 

□ 

b. editing 

□ 

□ 

□ 

□ 

c. storing 

□ 

□ 

□ 

□ 

d. retrieving video clips 

□ 

□ 

□ 

□ 

3. I am skilled at using Microsoft 
Office Power Point capabilities for 
creating user-friendly, user-choice 
navigation for online professional 
development products. 

□ 

□ 

□ 

□ 

4. *1 can articulate look fors in 
evaluating student proficiency in 
the practices of science and 
engineering. 

□ 

□ 

□ 

□ 
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5. 


6 . 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

I can explain how K-8 technology 
standards are relevant to my role as 
a teacher and as a STE teacher 
leader. 

□ 

□ 

□ 

□ 

I can explain how the practices of 
science and engineering and the 
Framework for K-12 Science 
Education are relevant to my role 

□ 

□ 

□ 

□ 


as a teacher and as a STE teacher 
leader. 


2. Is this your first or second year in STELP? First 


Second 
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Addendum to Survey - For New Teacher Leaders in Year 2 
Science Technology Engineering Leadership Program (STELP) 

Year 2, Oct 27, 2011 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. This is part of a series of surveys that will be given after each 
training session, thus our need to collect names; however your answers are strictly confidential 
(i.e. answers will not be linked to individual names for reporting). 

Name: 

School: 

1. How many years have you been in your current role (including the current year)? 


2. How many years have you been a classroom teacher (including the current year)? 


3. Do you have a degree or certification in science, technology, and/or engineering? 
□ Yes DNo 


4a. How many years have you taught Science in the classroom 
(including the current year)? 

4b. How many years have you taught Technology in the classroom 
(including the current year)? 

4c. How many years have you taught Engineering in the classroom 
(including the current year)? 

5. Have you ever participated in any kind of online training? □ Yes □ No 


5a. If yes, please describe the extent of your experience with online training. 
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6. Do you have any experience leading or developing professional development? 
I I Yes O No 


6a. If yes, did any of your experiences include leading or developing online professional 
development? 

I I Yes O No 


7b. If yes, please describe the extent of your experience leading or developing trainings. 


8. If you are an elementary teacher, to what extent is science integrated into the students’ 
schedule at your school? 

I | A lot ^Somewhat OA little QNot at all 

9a. If you are a middle school teacher, to what extent are math T and science teachers 
collaborating with each other on lesson planning and looking at student work? 

! | A lot ^Somewhat QA little QNot at all 


9b. If you are a middle school teacher, to what extent are technology education, math, and 
science teachers collaborating with each other on lesson planning and looking at student work? 

I I A lot [^Somewhat OA little ONot at all 


10. List any science activities that students are given the opportunity to participate in outside of 
the curriculum. 
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Survey Part II 

Science Technology Engineering Leadership Program (STELP) 

Year 2, Oct 27, 2011 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. This is the first in a series of surveys that will be given after each 
training session during year two of the program, thus our need to collect names; however, your 
answers are strictly confidential (i.e., answers will not be linked to individual names when 
reporting). 

Name: 

School: 

1. Please check your current position(s) (enter all that apply) 

| | Kindergarten Teacher O ES/MS Technology Education Teacher 

M Grade 1 Teacher O Staff Development Teacher 

| | Grade 2 Teacher O Special Ed Teacher (specify grades 

I i Grade 3 Teacher O Focus Teacher (specify grades + subject, 

M Grade 4 Teacher O Science Resource Teacher 

i j Grade 5 Teacher O MS Science Teacher (specify grades_ 

! 1 ES Science Teacher - lab or special (specify grades ) 

I I Other (please specify ) 


2. Please indicate any Leadership or Professional Development Roles you currently have: 


J 

J 

.) 
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Feedback on Today’s Training 


4. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 

Strongly Strongly 

Agree Agree Disagree Disagree 

a. 

The goals of today’s training were clear. 

□ 

□ 

□ 

□ 

b. 

The objectives of today’s training were 
met. 

□ 

□ 

□ 

□ 

c. 

The trainers were knowledgeable and 
well-prepared. 

□ 

□ 

□ 

□ 

d. 

An environment was created in which I 
felt comfortable taking risks (i.e., asking 
questions, expressing my ideas, working 
with unfamiliar content). 

□ 

□ 

□ 

□ 

e. 

Opportunities were provided for me to 
process and reflect upon the application 
of the knowledge and skills learned. 

□ 

□ 

□ 

□ 

f. 

My questions during the training today 
were answered adequately. 

□ 

□ 

□ 

□ 

g- 

As a result of today’s training, I have 
gained information and skills that will 

□ 

□ 

□ 

□ 


help me in this role. 


5. Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) 
with your comments or explanation/elaboration of your rating. 


6. What is the most important thing you gained from this training? 
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7. Do you have any suggestions for improving the training? 


Overall Feedback on the STELP Project 


8. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

The expectations for the professional 
development product are clear. 

□ 

□ 

□ 

□ 

The expectations for next steps are clear. 

□ 

□ 

□ 

□ 


9. Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) 
with your comments or explanation/elaboration of your rating.) 


10. Are there any challenges you are facing in your teacher leader role as you develop your 
professional development topic and online product? 
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11. Are there any additional suggestions you would like to give or resources that you think you 
will need to fulfill the teacher leader role as you develop your professional development product? 


Thank you for your help. 
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Post Training Survey 

Science Technology Engineering Leadership Program (STELP) 

Year 2, Dec. 6, 2011 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the second in a series of surveys that will be given after each training session during year 
two of the program, thus our need to collect names; however, your answers are strictly 
confidential. We may show verbatim answers to program staff, but no identifying information 
will be provided and answers will not be linked to individual names when reporting. 

Name: 

School: 

Team: (science practice number). 

Feedback on Today’s Training 


1. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 

Strongly Strongly 

Agree Agree Disagree Disagree 

a. The goals of today’s training were 
clear. 

□ 

□ 

□ 

□ 

b. The objectives of today’s training 
were met. 

□ 

□ 

□ 

□ 

c. The trainers were knowledgeable and 
well-prepared. 

□ 

□ 

□ 

□ 

d. An environment was created in which 





I felt comfortable taking risks (i.e., 
asking questions, expressing my 
ideas, working with unfamiliar 
content). 

□ 

□ 

□ 

□ 

e. Opportunities were provided for me 
to process and reflect upon the 
application of the knowledge and 
skills learned. 

□ 

□ 

□ 

□ 

f. My questions during the training 
today were answered adequately. 

□ 

□ 

□ 

□ 

g. As a result of today’s training, I have 
gained information and skills that will 

□ 

□ 

□ 

□ 

help me in this role. 
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2. Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) 
with your comments or explanation/elaboration of your rating.) 


3. What is the most important thing you gained from this training? 


4. Do you have any suggestions for improving the training? 
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Overall Feedback on the STELP Project 


5. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 


a. The expectations for the professional 
development product are clear. 


b. The expectations for next steps are 
clear. 


Strongly 



Strongly 

Agree 

Agree 

Disagree 

Disagree 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


6. Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) 
with your comments or explanation/elaboration of your rating.) 


7. Are there any challenges you are facing in your teacher leader role as you develop your 
professional development topic and online product? 


8. Are there any additional suggestions you would like to give or resources that you think you 
will need to fulfill the teacher leader role as you develop your professional development product? 


Thank you for your help. 
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Post Training Survey 

Science Technology Engineering Leadership Program (STELP) 

Year 2, Feb. 17, 2011 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the third in a series of surveys that will be given after each training session during year 
two of the program, thus our need to collect names; however, your answers are strictly 
confidential. We may show verbatim answers to program staff, but no identifying information 
will be provided and answers will not be linked to individual names when reporting. 

Name: 

School: 


Team: (science practice number) 

STE Concepts 


1. Please indicate how much you agree or disagree with the following items by checking the 

appropriate box. 

Strongly 



Strongly 


Agree 

Agree 

Disagree 

Disagree 

a. I can articulate look fors in 





evaluating student proficiency in 
the practices of science and 

□ 

□ 

□ 

□ 

engineering. 





b. I can explain how K-8 technology 





standards are relevant to my role as 
a teacher and as a STE teacher 
leader. 

□ 

□ 

□ 

□ 

c. I can explain how the practices of 





science and engineering and the 
Framework for K-12 Science 
Education are relevant to my role as 

□ 

□ 

□ 

□ 


a teacher and as a STE teacher 
leader. 
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Professional Development Training 

2. Please indicate how helpful you found the following items by checking the appropriate 
box. 

Not Not at 

Very Somewhat Very all 




Helpful 

Helpful 

Helpful 

Helpful 

a. 

How helpful was the feedback provided 
by the peer review in developing your 
online PD? 

□ 

□ 

□ 

□ 

b. 

How helpful was the training plan in 
developing your on-line PD? 

□ 

□ 

□ 

□ 

c. 

How helpful was today’s article 
discussion on Science standards in 
further understanding the goals of 

□ 

□ 

□ 

□ 


STELP? 


Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) with 
your comments or explanation/elaboration of your rating.) 


3. What is the most important thing you gained from this training? 


4. Do you have any suggestions for improving the training? 
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5. How useful would you find training for the following skills in one of the upcoming 
professional development sessions? Indicate usefulness by checking the appropriate box. 

Extremely Very A little Not at all 

Useful Useful Bit Useful Useful 

a. Using “Flip” style camera technology 

□ 

□ 

□ 

□ 

b. Using Windows Movie Maker 
Software 

c. Processing using Movie Maker 
software 

□ 

□ 

□ 

□ 

d. Editing using Movie Maker 

□ 

□ 

□ 

□ 

e. Storing using Movie Maker 

□ 

□ 

□ 

□ 

f. Retrieving video clips using Movie 
Maker 

□ 

□ 

□ 

□ 

g. Using Microsoft Office PowerPoint 
for online professional development 

□ 

□ 

□ 

□ 


products 

Development of Professional Development Products 


6. Please indicate how much you agree or disagree with the following items by checking 
the appropriate box. 

Strongly Strongly 

Agree Agree Disagree Disagree 

a. 

I feel good about the progress my 
teacher team has made so far. 

□ 

□ 

□ 

□ 

b. 

The expectations for what our teacher 
team is to accomplish for this project 
is clear. 

□ 

□ 

□ 

□ 

c. 

The expectations for next steps were 
clearly communicated. 

□ 

□ 

□ 

□ 

d. 

I feel good about the direction my 
team’s final product is headed in. 

□ 

□ 

□ 

□ 

e. 

I believe today’s session has helped 
with our teacher team’s progress as 
we develop our professional 

□ 

□ 

□ 

□ 


development topic and product. 
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Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) with 
your comments or explanation/elaboration of your rating.) 


7. How would you describe your team’s progress on your professional development product? 

□ Not as far along as we would like; running behind our planned time line 

□ Right on schedule 

□ Ahead of our planned time line 

8. Are there any challenges you are currently facing in your teacher leader role as you develop 
your professional development topic and online product? 


9. Are there any additional suggestions you would like to give or resources that you think you 
will need to fulfill the teacher leader role as you develop your professional development product? 
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Post Training Survey 

Science Technology Engineering Leadership Program (STELP) 

Year 2, April 26, 2012 

As part of your involvement in the Science, Technology, and Engineering Leadership Program 
(STELP), we will be asking you to provide information and feedback about the program. Your 
input is very important to the evaluation of the program and it will help guide administrators in 
planning and implementation. 

This is the fourth in a series of surveys that will be given after each training session during year 
two of the program, thus our need to collect names; however, your answers are strictly 
confidential. We may show verbatim answers to program staff, but no identifying information 
will be provided and answers will not be linked to individual names when reporting. 

Name: 

School: 


Team: (science practice number). 

Professional Development Training 


1 . Please indicate how helpful you found the following items today by checking the 
appropriate box. 



Very 

Helpful 

Somewha 
t Helpful 

Not Very 
Helpful 

Not at all 
Helpful 

How helpful was the feedback provided 
by the peer review in developing your 

□ 

□ 

□ 

□ 

online PD? 

How helpful was the STE instructional 
specialist assigned to your group? 

□ 

□ 

□ 

□ 


2. Please elaborate how helpful the peer review was to your product development (see Q2a), 
including the way in which the peer reviews were grouped this session. 
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3. What is the most important thing you gained from this professional development session? 


4. Do you have any suggestions for improving the professional development sessions? 


Development of Professional Development Products 


5. 

Please indicate how much you agree or disagree with the following items by checking 

the appropriate box. 

Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. 

I feel good about the progress my 
teacher team has made so far. 

□ 

□ 

□ 

□ 

b. 

The expectations for what our teacher 
team is to accomplish for this project 
is clear. 

□ 

□ 

□ 

□ 

c. 

The expectations for next steps were 
clearly communicated. 

□ 

□ 

□ 

□ 

d. 

I feel good about the direction my 
team’s final product is headed in. 

□ 

□ 

□ 

□ 

e. 

I believe today’s session has helped 
with our teacher team’s progress as 
we develop our professional 

□ 

□ 

□ 

□ 


development topic and product. 
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6. Please elaborate on any of the statements above. (Indicate the corresponding letter item(s) 
with your comments or explanation/elaboration of your rating.) 


7. How would you describe your team’s progress on your professional development product? 

□ Not as far along as we would like; running behind our planned time line 

□ Right on schedule 

□ Ahead of our planned time line 

8. Are there any challenges you are currently facing in your teacher leader role as you develop 
your professional development topic and online product? 


9. Are there any additional suggestions you would like to give or resources that you think you 
will need to fulfill the teacher leader role as you develop your professional development product? 


Thank you! 
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Survey 

Science Technology Engineering Leadership Program (STELP) 

Year 2, May 18, 2012 

This is the last in a series of surveys this year that will be given after each training session during 
year two of the program, thus our need to collect names; however, your answers are strictly 
confidential (i.e., answers will not be linked to individual names when reporting). 

Name: 

School: 


Please indicate how much you agree or disagree with the following items by checking the 


appropriate box. 

Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

1. I am skilled in using “Flip” style 
camera technology for capturing 
student learning. 

□ 

□ 

□ 

□ 

2. I am skilled at using Windows 
Movie Maker software for. . . 

a. processing 

□ 

□ 

□ 

□ 

b. editing 

□ 

□ 

□ 

□ 

c. storing 

□ 

□ 

□ 

□ 

d. retrieving video clips 

□ 

□ 

□ 

□ 

3. I am skilled at using Microsoft 
Office PowerPoint capabilities for 
creating user-friendly, user-choice 

□ 

□ 

□ 

□ 


navigation for online professional 
development products. 


4. I can articulate look fors in 





evaluating student proficiency in 
the practices of science and 





engineering. 

□ 

□ 

□ 

□ 
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5. I can explain how K- 8 technology 
standards are relevant to my role as 
a teacher and as a STE teacher 
leader. 

6. I can explain how the practices of 
science and engineering and the 
Framework for K-12 Science 
Education are relevant to my role as 
a teacher and as a STE teacher 
leader. 


Strongly 



Strongly 

Agree 

Agree 

Disagree 

Disagree 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


Development of Professional Development Products 


7. Please indicate how much you agree or disagree with the following items by checking the 
appropriate box. 



Strongly 

Agree 

Agree 

Disagree 

Strongly 

Disagree 

a. I feel good about the progress my 
teacher team has made. 

□ 

□ 

□ 

□ 

b. The expectations for next steps were 
clearly communicated. 

□ 

□ 

□ 

□ 

c. I feel good about our team’ s final 





product/the direction my team’s final 

□ 

□ 

□ 

□ 


product is headed in. 


8. How would you describe your team’s progress on your professional development product? 

□ Not as far along as we would like; running behind our planned time line 

□ Right on schedule 

□ Ahead of our planned time line 


Please use this space any additional comments: 


Thank you for your assistance! 
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Appendix B: Program Evaluation Unit Project Rubric 


STELP Online Professional Development Project RUBRIC 
Summary of 7 modules 


Research Claims 

Was there cited research, outside the practice, included in the module? 

Was the practice cited? 

Criteria 

Exemplary 

(3) 

Proficient 

(2) 

Not Proficient 
(1) 

Score 


Organization 

Sequence 

The sequence of information 
is logical and intuitive 
*Does it build up? 

The sequence of information 
is somewhat logical 
*Does it build up? 

The sequence of 
information is not logical. 
*Does it build up? 


Description of 
Practice 

Presentation includes a clear 
and comprehensive 
description of the practice as 
defined in the Framework 
(i.e. definition plus further 
description or explanation) 

Presentation includes a 
nominal description of the 
practice as defined in the 
Framework, but not enough 
information is provided for 
clear understanding, (i.e., just 
given the definition) 

Presentation does not 
include a description of 
the practice as defined in 
the Framework 


Use by scientists 
and engineering 

Presentation includes how the 
practice is used by scientists 
and engineers (with examples 
of “how”) 

Presentation somewhat 
includes how the practice is 
used by scientists and 
engineers (no examples or 
partially explained) 

Presentation does not 
include how the practice 
is used by scientists and 
engineers 


Impact on Student 
Learning 

Presentation includes the 
impact of the practice on 
student learning (could be 
links to research; could show 
examples); connects with 
practice, shows impact on 
learning 

Presentation somewhat 
includes the impact of the 
practice on student learning 
(could be links to research; 
could show examples); 
doesn’t connect with practice 
or doesn’t show impact on 
learning or only partially 

Presentation does not 
address the impact of the 
practice on student 
learning 


Classroom 

exemplars 

Presentation includes 
classroom exemplars of the 
practice (multi-grade level); 
enough examples that a 
teacher in any grade could 
learn something relevant? 

Some Classroom exemplars 
of the practice but not enough 
that teachers in all grades 
would learn relevant 
information 

Presentation does not 
include classroom 
exemplars of the practice 
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Criteria 

Exemplary 

(3) 

Proficient 

(2) 

Not Proficient 
(1) 

Score 

Instructional 

strategies 

Presentation includes 
instructional strategies for 
supporting students in 
developing practice. 
Example: some videos of 
experiences; sample of 
student products; worksheet 

Presentation includes 
Instructional strategies for 
supporting students in 
developing practice in only a 
limited wav (e.g., shows 
classroom activity but doesn’t 
explain strategy or make the 
link to developing practice). 

Presentation does not 
include Instructional 
strategies for supporting 
students in developing 
practice. 



Design 

Multimedia 

Project is truly multimedia 
and interactive, and 
Is there a variety? (i.e., 
videos, pictures, slides) 

Proiect contains one or two 
media choices for viewer. The 
design is primarily linear. 

Project contains no 
choices. The design is one 
dimensional. 


Navigation 

Presentation contains well- 
designed navigational choices 
for viewer. 

*Look at this lx in sequence; 
2 nd time jumping around 

Presentation contains 
somewhat well-designed 
navigational choices for 
viewer. 

Presentation contains no 
well-designed 
navigational choices for 
viewer ( i.e., linear with 
no choices or not well 
designed). 


Technical Issues 

All technical functions seem 
to work correctly 

Some technical functions do 
not work correctly (i.e., links, 
loading, etc.) 

Many technical functions 
do not work correctly 
(i.e., links, loading, etc.) 


Visual 

Presentation 

All Visual Presentations (i.e., 
text, graphics) are of good 
quality (can read, graphics 
clear) 

Some Visual Presentations 
(i.e. text, graphics) are of poor 
quality (hard to read, graphics 
not clear) 

Much of the Visual 
Presentation (i.e., text, 
graphics) is of poor 
quality (hard to read, 
graphics not clear) 


Audio Presenation 

All Audio Presentations (i.e., 
narrator, video clips) are of 
good quality (i.e., can hear, 
clear) 

Some of the Audio 
Presentations (i.e., narrator, 
video clips) are of poor 
quality (i.e., cannot hear, not 
clear) 

Much of the Audio 
Presentations (i.e., 
narrator, video clips) are 
of poor quality (i.e., 
cannot hear, not clear) 


Artifacts** 

All the photographs, 
graphics, sound and/or video 
align appropriately and 
enhance the content as well 
as create interest. 

*Is there a conection to the 
content? 

Some of the photographs, 
graphics, sound and/or video 
enhance the content and 
create interest. 

The photographs, 
graphics, sound and/or 
video are inappropriate 
for the content and do not 
create interest. 



Program Evaluation Unit 


47 


Evaluation of STELP, Year Two 



Montgomery County Public Schools 


Office of Shared Accountability 


Criteria 

Exemplary 

(3) 

Proficient 

(2) 

Not Proficient 
(1) 

Score 


Check for Understandin 

§ 

Opportunities 

Session consists of multiple 
opportunities for a viewer to 
process, reflect on, and apply 
new information as well as to 
evaluate learning. 

Could be quizmaker; choice 
link with explanation; 
reflection journals 

Session consists of one 
opportunity for a viewer to 
process, reflect on, and apply 
new information or to 
evaluate learning. 

There are no 

opportunities for a viewer 
to process, reflect on, and 
apply new information or 
to evaluate learning. 


Connection 

All Check for Understandings 
had a clear connection to 
what was presented 

Some Check for 
Understandings had a clear 
connection to what was 
presented or had a partial 
connection 

No Check for 
Understandings had a 
clear connection to what 
was presented 

N/A - No Check for 
Understanding 


Answer Review 

All Check for Understanding 
questions were 
reviewed/gave correct 
answers 

Some Check for 
Understanding questions were 
reviewed/gave correct 
answers or had partial 
answers/review 

No Check for 
Understanding questions 
were reviewed /gave 
correct answers - i.e., 
never find out results 

N/A - No Check for 
Understanding 


Preassessment 

Connection 

All preassessment questions 
are revisited or addressed in 
the presentation 

Some preassessment 
questions are revisited or 
addressed in the presentation 

No preassessment 
questions are revisited 

N/A - There are no 
preassessments 


Goals/Outcomes 

All goals or outcomes stated 
in the presentation are 
covered 

Some goals or outcomes 
stated are covered/Goals or 
outcomes are partially 
covered 

Goals or outcomes stated 
are not covered 

N/A - No Goals or 
outcomes were given 
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Appendix C: STE Staff Interview 

1. Were you involved in both year 1 and year 2 of the Howard Hughes STELP grant? 

2. What has been your role(s) in this year’s STELP project? 

a. Probe if needed: working with a team? Editing products? Training tasks? Etc. 

3. How often have you met with your assigned group? Did you meet with other groups? 

4. What depth of assistance did you provide your group? How much ownership do you 
have in the product? 

a. Probe if needed: did you assist only if needed/by request? Did you guide or 
actually take a task to do yourself? Etc. 

5. How did you feel about the process of the trainings and product development this year? 

a. What were some good approaches or steps taken? 

b. What were some challenges or not good approaches taken? 

6. Did you face any challenges yourself as you fulfilled your role? 

7. How did you feel your team worked together? 

a. Probe if needed: did they have conflicts? Hard to get together? 

8. What challenges did your team face when working on their professional development 
project? 

9. So, how do you feel about the final professional development project they turned in? 

10. What are the next steps for the on-line products? 

11. How do you think you all will communicate to the wider audience of teachers to get them 
to take the online professional development? 

12. What would you like to see implemented next year during the third year of this project? 

a. Probe if needed: What would you like to see again next year? What would you 
like to see differently? Any additional activities/steps you’d like to see happen 
next year? 

Extra Question (if time allows) 

What do you think is the most important thing that teachers who view the professional 
development projects online will gain from them? 
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Appendix D: STELP Training Plan Peer and STE Specialist Review 

February 17, 2012 

For each item, circle 0, 1, or 2. Do not leave any items unanswered. (0 = No; 1 = Some 
evidence; 2 = Yes) 


Reviewer: Practice Reviewed: 


Training Plan Component 

Score 

Recommendations for Improvement 

Outcome(s) 

The stated outcome identifies the scientific or 
engineering practice that a viewer of the PD will 
explore. 

0 1 2 


Outcome(s) 

The stated outcome identifies what a viewer of 
the PD will know and be able to do as a result of 
completing the online PD. 

0 1 2 


Identifying Mastery 

The training plan includes a detailed description 
of how mastery of the objective will be 
determined and any needed artifacts/resources. 
For example, “Participants will demonstrate an 
understanding of asking questions and defining 
problems through video and/or picture and/or 
case study scenario identifications.” 

0 1 2 


Engage 

The training plan clearly describes a learning 
task, and needed artifacts/resources, for viewers 
of the online PD which focuses viewer’ s 
attention, stimulates their thinking, and accesses 
their prior knowledge. 

0 1 2 


Explain 

The training plan clearly describes at least two 
opportunities, and needed artifacts/resources, 
for viewers of the online PD to process, reflect 
on, and explain their thinking. 

0 1 2 


Explore 

The training plan clearly describes several 
opportunities, and needed artifacts/resources, 
for viewers of the online PD to have 
experiences with the concepts and ideas of the 
scientific and engineering practice. 

0 1 2 


Evaluate 

The training describes strategies and tools, and 
needed artifacts/resources, for pre-assessment, 
on-going checks-for-understanding, and an end- 
of-session assessment. 

0 1 2 
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Training Plan Component 

Score 

Recommendations for Improvement 

Extend/Elaborate (Optional) 

The training plan describes learning tasks, and 
needed artifacts/resources, in which the 
viewers of the online PD are able to apply their 
new knowledge and skills to new, but similar 
situations. 

0 1 

2 


Overall 

The training plan content about the specific 
scientific and engineering practice is accurate 
and adequately addressed within the session. 

0 1 

2 


Overall 

All puiposed artifacts (videos, pictures, student 
work samples, etc.) are aligned appropriately to 
the content. 

0 1 

2 


Overall 

The sequence of information is logical and 
intuitive. 

0 1 

2 



Additional comments and feedback: 
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Appendix E: Training Session Schedule, Attendance, and Outcomes 


Science, Technology, and Engineering Leadership Program: 
Training Sessions 


Date of training session 


September 20, 2011 
For participants new to 
the program 


October 27, 2011 
PD Session #1 


December 6, 201 1 
PD Session #2 


Session Learning Outcomes and Tasks 

• Articulate the vision for science, technology, and 
engineering in MCPS 

• Identify student proficiencies in science and 
engineering 

• Facilitate use of video recording technology in 
classroom instruction 

• Use Windows Movie Maker and Microsoft 

PowerPoint to create instructional resources 

• Articulate the vision for science, technology, and 
engineering in MCPS 

• Explain how the MCPS science, technology, and 
engineering vision aligns to the Framework for K-12 
Science Education and the Next Generation Science 
Standards (NGSS) 

• Describe how the strands for science and engineering 
proficiency align to the Framework for K-12 Science 
Education 

• Evaluate student proficiency in science and 
engineering through the lens of target proficiencies 

• Articulate next steps in STELP online professional 
development product work 

Additionally: 

• Reviewed and provided written feedback on year one 
products 

• Met assigned team and determined team roles and 

next steps 

• Use video cameras for capturing students engaged in 
scientific and engineering practices 

• Explain the 201 1-2012 STELP tasks and time lines 

• Describe the connection between the strands for 
science learning and the scientific and engineering 
practices 

• Identify outcomes for professional development 
projects and develop a training plan 

• Explain the vision for the NGSS; included team 
work and guest speaker, Vice President of Achieve, 
who talked about the NGSS framework, practices 
and standards 


Number 

attending 


37 of 47 


38 of 47 


40 of 47 
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Number 


Date of training session 

Session Learning Outcomes and Tasks 

attending 

February 17, 2012 
PD Session #3 

• Presented training plan drafst for review 

• Reviewed training plan drafts of peers and provided 
recommendations for upgrades 

• Articulated next steps in STELP online professional 
development product work 

Additionally: 

• Discussion on NGSS 

• Team work on training plan and product (video clips, 
PowerPoint) 

31 of 47 

April 26, 2012 
PD Session #4 

• Presented PowerPoint drafts for review 

• Reviewed PowerPoint drafts of peers and provided 
recommendations for upgrades 

• Articulated next steps in STELP online professional 
development product work 

Included: 

• Received new video cameras 

• Team project work 

31 of 47 

May 18, 2012 
PD Session #5 

• Team project work session only 

n/a 

May 24, 2012 
May 29, 2012 

June 7, 2012 

• Optional days for participants to work on products at 
computer lab facility 

N/A 

N/A 

N/A 

June 8, 2012 


N/A 

June, 2012 

Additional work session offered as needed 

N/A 


Note:. Precise attendance counts for each session are unknown. The attendance counts are based on the number of 
feedback surveys returned; therefore, attendance may be underestimated. N/A designates not available. 
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Appendix F: Participants’ STELP Reflection 
2011-2012 STELP REFLECTION 

MCPS STEM Vision — All students achieve full science, technology, engineering, and math (STEM) 
literacy through seamlessly integrated instruction that is project/problem and standards-based. STEM 
literate students are critical thinkers who are able to solve non-routine problems in a globally 
competitive society. The primary goal of STELP is to build a cadre of STE teacher leaders to support 
the MCPS STEM vision by designing and delivering online PD about student proficiency in science 
and engineering. Teachers selected for this program must be committed to the MCPS vision and the 
STELP goal as evidenced in the work they do with students and teacher colleagues. Directions: 
Complete each item below. Be sure to provide specific evidence of how this program has supported 
your professional growth in understanding and implementation of STEM instruction. 


* Required 
Top of Form 

1. Describe how the STELP professional development on science and engineering proficiencies 
(Ready, Set, SCIENCE!) and scientific and engineering practices (Framework for K-12 Science 
Education) has impacted your instruction? (e.g., How have you changed your teaching to address 



2. Describe how the STELP professional development on science and engineering proficiencies 
(Ready, Set, SCIENCE!) and scientific and engineering practices (Framework for K-12 Science 
Education) has impacted your work with colleagues or events at your school? (e.g., How have 
you shared what you have learned about the science and engineering practices with colleagues?, 
How have you used the science and engineering practices to upgrade science events at your 
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3. PRIOR to participating in STELP, what was your skill level? Please rate you level of proficiency 
with each of the following. * 1 = Basic - some knowledge but would need significant help; 2 = 
Proficient - knowledgeable but might need some assistance; 3 = Advanced - can do on my own. 



1 

2 

3 

Movie Maker 

□ 

□ 

G 

PowerPoint for online learning 

(multimedia and navigational 
options) 

c 

G 

G 

Writing 5E professional 
development training plans 

c 

G 

G 


4. As a result of participating in STELP, what is your skill level? Please rate you level of 
proficiency with each of the following. * 1 = Basic - some knowledge but would need significant 
help; 2 = Proficient - knowledgeable but might need some assistance; 3 = Advanced - can do on 
my own. 


1 2 3 


Movie Maker 

G 

G 

G 

PowerPoint for online learning 




(multimedia and navigational 
options) 

G 

G 

G 

Writing 5E professional 
development training plans 

C 

G 

G 


4a. If you rated yourself 1 (Basic), please explain what prevented you from obtaining more skill and 
what would help you to improve 




1 


IT 1 
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Jst your contributions to your team's project. 


1 


jU 


A 


6. Add any other comments about how you have benefited as a result of your participation in the 



7. Would you like to continue in STELP? Explain how you think you would continue to grow 
professionally if you are to continue in this program. The work for next year will be a 
continuation of building online PD about classroom instruction that models the scientific and 


engineering practices as well as the Crosscutting Concepts out 


ined in the Framework. 


[JJ 


A 


8. Name * Your name is required in order for you to be paid for this year's work. 


Submit 


Bottom of Form 
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